Deriving Vertical Orientation from Anchor-Based Assessments

Modern search engine result pages are becoming more and more heterogeneous. This is mostly achieved by adding a special vertical results on top of traditional “general web” results. These results usually come from special sources (verticals) and the choice of verticals is different for different queries. Vertical orientation is an important value that quantifies the user’s need of having results from a particular vertical (e.g., News, Blogs, Video) on a search engine result page (SERP). It is used not just for selecting relevant verticals and positioning them on a SERP, but also for building vertical-aware click models and evaluating aggregated search performance. In this paper we propose a way to accurately estimate vertical orientation from a limited amount of human assessments. We describe an intuitive procedure of collecting human ratings and show how these ratings can be converted to real-valued estimates of vertical orientation and further extrapolated to unseen queries with the help of machine learning.